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Results of Iterative Standards-Setting 
Procedures for a Performance-Based 
System for Renewable Certification 

Overview 

Tlie Louisiana System for Teaching and learning Assessment and Revie\ ' (STAR) (Ellett, 
Loup & Cliauvin, 1990) lias been developed and piloted throughout Louisiana for the Louisiana 
Department of Education during the past two years to meet requirements of the Louisiana 
Teaching Internship Law (1984) and the Children First Act (1988). Following the lead of many 
other stales, Louisiana is implementing an induction process for new teachers that includes 
comprehensive, classroom- based assessments to be used as a basis for structuring support 
programs for continuing professional development. The ulthnate goal of the Louisiana Teaching 
Internship Program is to provide new teachers with successful experiences and support to assist 
them in their professional development during llie early years of employment. 

The Children First Act provides for on-the-job assessments of all experienced teachers in 
Louisiana for the purpose of obtaining a professional, renewable certificate with a maximum 
validity period of five yeiu-s. This Act mandates that all Louisiana teachers (excluding interns) 
be evaluated with a standaid system for the purpose of validating and/or renewing tlie 
professional teaching credential. The STAR and the accompanying set of assessment procedures 
will be used to make these certification decisions. Star assessment data will also be used to 
provide comprehensive, diagnostic information to teachers about the quality and effectiveness of 
teaching and learning to assist them in their continuing professional development. The STAR 
is being implemented statewide for the first time in Louisiana during the 1990-1991 school year. 
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Because of the extensive involvement of educators at all levels in developing and 
implementing these new assessment programs, the STAR has implications that go beyond 
meeting the requirements of new Louisiana laws. For example, the STAR has been used as a 
framework for examining selected elements of teacher education programs and as a basis for 
Inservice education and staff development. The STAR has also proven useful in conducting 
classroom-based research on effective teaciiing and learning and in understanding elements of 
teachers' professional experiences, expertise, and perspectives on reflective practice (Ellett. 
1990a). 

The STAR is a comprehensive, on-the-job assessment instniment and set of procedures 
designed to make inferences about the quidity of teaching and learning in Louisiana's classrooms. 
The STAR is grounded in some fifteen years of research and development in designing and 
implementing large-scale teacher evaluation systems in other states (Elicit, Garland & Logan, 
1987). However, the STAR document and the assessment processes extend prior generations of 
teacher evaluation instruments to include important concerns with classroom context variables 
and student learning as well (Ellett, 1990a). 

A variet> of ongoing research and development studies designed to investigate the validity, 
reliability and implemcntability of the STAR have been completed (Ellett, 1990b; LaMaster, 
Tobin & Bowen, 1990; Teddlie, Ellett & Naik, 1990; Chauvin, 1990; Loup, Ellett & Chauvin, 
1990; Ellett, Loup, Chauvin & Naik, 1990; Elicit, Chauvin, Loup, & Naik. 1990), or in 
process. The results of these studies are encouraging and typically they support the STAR as a 
professionally endorsed, valid and reasonably reliable system designed to fulfill the purposes of 
the Louisiana Teaching Internship and Statewide Teacher Evaluation efforts. 
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A very iniporlant research and development concern with the STAR is establishing 
performance standards for making teacher certification decisions. This paper describes the results 
of an initial STAR standards-setting task completed by a select group of Louisiana educators 
during tiic late spring of 1990. It also includes a brief review of subsequent events which 
culminated in a set of final perfonnaiice standards for the STAR in the fall of 1990. Issues, 
procedures, and "caudons" relative to standiu:ds-setting are discussed. 

Perspectives on Standards-Setting 

Setting performance standards for systems like the STAR is not an easy task. It requires that 
educators consider many different kinds of information as they proceed and that they are 
reasonably aware of the consequences of their professional judgements relative to the 
perfoniiance standiu-ds they recommend. Ultimately, setting standards is a "lunnan" concern 
because the task requires that professional judgements be made at many points in the process. 
As die saying goes, "computers don't set standards...humans do." The key question becoiues how 
do we systematize a process for making iiifonned professional judgements that is scientifically 
based, replicable and results in standards that are reasonably valid and reliable (1985 Standards 
for Educational and Psychological Testing)? Jaeger (1990) poses a number of issues to be 
considered: number, qualification, selection and training of persons to recommend and set 
standards, what tlxy will do and how data will be collectea, analyzed and reported. 

While the standaid-seiting liteniture includes recommendations on some of these issues, 
most have been made based on paper-and-pencil tests rather th-»n on classroom observadon 
procedures. Many are based on logic rather than empirical findings. Selecting individuals that 
are: 1) highly informed ("experts"); 2) professionally committed tp take the task seriously; and 
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3) that represent a variety of "stakeholder" perspectives, is a primary concern. Even though 
standards setting is ultimately a human concern, providing standards setters with different kinds 
of information is also critical to setting standards lliat are reasonable, equitable and professionally 
credible. Therefore, as standards are set, understanding various kinds of information pertaining 
to: 1) assessment processes and procedures; 2) program implementation policies and practices; 
3) data aggregation rules for making decisions; and 4) results of research studies, is also an 
important concern. 

The 1990-1991 STAR is designed to assess and make inferences about the quality of teaching 
and learning on sets of assessment indicators defining 21 Teaching and Learning Components. 
The assessment indicators and components comprising the STAR have been professionally 
endorsed by teachers and other selected educators throughout Louisiana in two separate research 
studies, as "essential" for both certification and the enhancement of student learning. The STAR 
is more than a single "test," and it requires consideration of multiple perfonuance standards for 
multiple, essential elements of effective teaching and learning. Thus, standards-setting 
considerations and activities with a system like the STAR are much more complex than those for 
more Uaditional evaluation instruments used for teacher certification that requu-e that single 
(whole score) standards be set (e.g., a single score for the National Teacher Examination). 

The STAR has been piloted statewide in Louisiana for the past two years (1988-1990). Pilot 
activities during 1989-1990 included certification of the principal and a master teacher as STAR 
assessors in every public school building in Louisiana. In addition, a variety of other educators 
(e.g., college faculty, assistant principals, parish supervisors, LDE personnel) participated in the 
STAR professional development program to certify assessors. As part of pilot activities during 
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1989-1990, a large number of STAR assessments (some 6,000) were completed and these 
assessments involved teachers, students and classrooms in every parish in Louisiana. Thus, there 
was a considerable amount of information available about characteristics of effective teaching and 
learning in Louisiana's classrooms. This information was analyzed in a variety of ways and the 
ensuing results were used by Louisiana educators as tiiey considered an initial set of STAR 
perfonnancc standards. However, at the time initial standards were set, no infonnation about 
performance on llie STAR was available that had been collected under the "real" and "high 
stakes" conditions of certification. These data have only become available as the STAR is 
implemented for certification statewide for the first time during the 1990-1991 school year. 

The results of the inidal standards-setting task which is the primary focus of this report were 
to be considered only tentative "benchmarks" because STAR perf"ormance distributions are likely 
to change under "real" assessment conditions pertaining to certification. It was intended that 
initial performance standards be revisited when additional STAR data was collected during Uie 
fall of 1990 and tlie spring of 1991. With assessment systems similar to the STAR, changes ir. 
perfomiance distributions are known to affect the dependability of certification decisions and to 
change misclassification probabilities from those established under "pilot" conditions (Capie & 
Ellett, 1982). Typically, such performance distributions shift toward somewhat higher scores 
when assessment data are collected under "real" conditions. Therefore, the initial performance 
standards were set with the idea that they should not be considered final by any means. Instead 
they represented a set oC reasonable "benchmarks" for teachers to consider as they began tiie 
STAR assessment process for renewable certification during the fall of 1990. 
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The Initial STAR Standards-Setting Tasic 

An initial standards-setting worl<sliop witJi Louisiana educators to recommend initial 
performance expectations for the STAR was held in June, 1990 in New Orleans, Louisiana. The 
purpose of this workshop was to provide a highly informed ("expert") group of Louisiana 
educators witli the results of STAR pilot research studies (1988-199U) to be used as critical 
infonnation for making initial STAR perfomiance standards recommendations for the Louisiana 
Teaching Internship and the Statewide Teacher Evaluation Programs. In addition, the workshop 
served as a forum for tlie presentation and discussion of critical professional and program policy 
and implementation issues that pertained lo standards recommendations. 
Standards-Setting Panelists 

Consistent with recommendations of Hambleton (1978) and Shepherd (1980) on the use of 
several types of judges, a panel of 47 educators from various regions of Louisiana was nominated 
by LTIP and LTEP regional, STAR professional development program coordinators giving 
consideration to two essential concerns: 1) knowledge and expertise in the STAR and the LTIP 
and LTEP programs; and 2) a reasonable balance among panel members relative to position of 
employment, ethnicity, gender and other key factors. In selecting panel members, an attempt was 
made to assure that the majority of panelists were regular classroom teachers. All panelists 
nominated/selected had extensive preparation as STAR assessors and many had served during the 
1989-1990 pilot us program assistants in the STAR professional development program to certify 
assessors. 

Table 1 presents a summary of demographic information on tlie standards-setting panelists 
that pai'ticipaled in the initial worksliop activities. (Table 1 and subsequent tables are in 
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Appendix A.) As can be seen in the table. 67.4% of the panelists were female, and 39.1% were 
black. The majority of panelists were regular classroom teachers (56.5%) and approximately 
17.5% were school administrators. Slightly less than one fourth of the panelists (23.9%) were 
college faculty, parish supervisors and LDE personnel. The majority of panelists represented 
elementary school levels (41.9%) and over half (56.5%) possessed plus tliirty. specialist or 
doctorate degrees. The average years of experience as an educator for these panelists was 19 
years, with almost all of this experience in public schools. The average years of teaching 
experience for the teacher panelists was 18.28 years. Four of these panelists were members of 
the LDE perfonnance standards, policy recommendaUons committee. Four of these panelists were 
LSU LTIP and LTEP regional coordinators of the professional development program to certify 
STAR assessors. Thus, these panelists represented a highly experienced, educated and reasonably 
balanced group of "stakeholders" with considerable knowledge of the STAR assessment 
document, assessment procedures, legal bases- of LTIP and LTEP. and existing program 
implementation procedures and policies. 

The LTIP and LTEP Project Director and three LSU project rcordinators organized and 
served as leaders for t!ie standards-setting workshop. The outside consultant for the workshop 
design was Dr. Richiu-d Jaeger. College of Education. University of North Carolina at 
Greensboro. 

Standards-Settings Task(s) 

The standards-setting process, adapted from the work of Jaeger (1990), was an "iterative" one 
that occurred over three and one-half days of intensive workshop activity. It included providing 
panelists with normative information as recommended by Hambleton and Eignor (1980), Jaeger 
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(1982, 1989) and Shepard (1980), and allowing judges to discuss and reconsider their 
recommendation (Jaeger. 1978. Brandstalter. Davis, and Stockcr-Kreichgauer. 1982). In this type 
of procedure, panelists are provided with successively more and more critical infonnation for 
discussion and individual performance standards recommendations are made. Data arc then 
tabulated and summarized, and group discussion of results, concerns and pertinent issues follows. 
Additional recommendations then made by each panelist and these are again tabulated, 
summarized and shared with the group for additional discussion and so on. At some point, a 
final recommendation is then made relative to an appropriate performance standard for each "test" 
or variable under consideration. A set of iterative procedures was completed serially for each 
STAR Teaching and Learning Component. 

In this workshop, panelists were first provided with an extensive review of the STAR 
document to clarify understandings of assessment indicators and Teaching and Learning 
Components since there was considerable variation in the time of the year in which each had 
completed the STAR professional development program to certify assessors. This review 
required almost one full day. Dunng the first part of the second day of the workshop, panelists 
received a lecture/discussion on issues pertaining to the reliability and validity of the STAR using 
pilot research results, and examples of the relationship between the reliability of the STAR and 
the "dependability" of the STAR for making classification decisions for certification. Tliis latter 
informaaon pointed out the relationsliip between the reliabilities of various STAR Teaching and 
Learning Components and the possibilities of making "false positive" and "false negative" 
decisions. The remiuning two workshop days were spent in small and large group activities 
reviewing STAR research data and iteratively making recommer dadons for performance 
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standards and discussing results and pertinent issues. 

At the end of llie last day. panelists made a final recommendation for a performance standard 
for each STAR Teaching and Learning Component. They also made a "voice vote" 
recommendation for a perfonnance expectation for intern te?.hers and a "one-sliot" 
recommendation for a "superior" performance standard for each STAR Teaching and Learning 
Component for teachers who might be considered for tlie Louisiana Mode' Career Options 
Program as specified in the Children First Act. As a final acavity. panelists were asked to 
carefully review the assessment indicators for eacli STAR Teaching and Learning Component and 
to make recommendations about any assessment indicator that they believed to be so "essential" 
that being judsed as "ccnsistenUy unacceptable" should result in an "Unacceptable" decision for 
the entire component. 

Each panelist was provided with a workshop manual that included workshop objectives, rules 
and schedules, response forms for recording judgements, participant evaluation forms for various 
workshop acdvities and for recording concerns and STAR research results information. Tlie key 
research information to be considered by each panelist in making recommendations was as 
follows: 

1. descriptive statistical summaries of STAR assessment indicator and Teaching and 
Learning Component data derived from 1989-1990 STAR assessments completed in 
5.473 classrooms/lessons representing every parish and virtually every public school 
building in Louisiana 

These summaries showed the percentages of "Acceptable" and "Unacceptable" performance 
judgements compiled by Louisiana educators during the 1989-1990 pilot year for elementary. 
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secondary and total classroom groups, These data were discussed as representative of "routine, 
daily perfonnances" in Louisiana's classrooms and cautions were emphasized about predicted 
shifts in perfonnance distributions under "high stakes" assessment conditions for cerafication 
during the fall of 1990. This large number of assessments had been completed by principals, 
master teaciiers and other educators, drawn from every parish in Louisiana, as part of STAR 
assessor certification requirements. 

2. graphs depicting the frequency and cunuilauve percentages of "Acceptable" decisions 

for assessment indicators and "Mastery" scores for each STAR Teaching and Learning 

Component 

These graphs provided punelists with information about various percentages of "Acceptable" 
decisions for various numbers (sets) of indicators from the pilot STAR assessments. They also 
depicted the percentages of teaciiers with "mastery" scores witliin various, possible score ranges. 
Considered collectively, these graphs provided panelists willi some information about the possible 
percentages of teachers that would "pass" or "fail" at various "cut points." 

3. summaries of generalizability (reliability) coefficients for STAR Teaching and 
Learning Components from the 1988-1989 and 1989-1990 pilot research studies 
These coefficients provided panelists with some information about how much they could "tmst" 
their performance standards recommendations in view of making potentiid "false positive" and 
"false negative" decisions. To simplify matters, panelists were iiifomied that "all other factors 
considered, the higher the 'G' coefficient, the greater is the 'trust' that can be placed in a 
recommended performance standard." 

Sample copies of descriptive statistical summaries for STAR Teaching and Learning 
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Coniponeiiis and ussessiiient indicators, percentages and cumulative frequency graphs, and 
gencralizability coefficients provided to panelists are shown in Appendix B. A set of information 
like the set shown in Appendix B was provided for each panelist for each Teaching and Learning 
Component in STAR Pcrfomiance Dimensions II (Classroom and Behavior Management). Ill 
(Learning Environment) and IV (Enhancement of Learning). Similar information did not exist 
for STAR Performance Dimension I (Preparation. Planning and Evaluation). However, this 
dimension was thoroughly studied aiid discussed by panelists in view of the results of two. small- 
scale, qualitative sluaies of teachers' preparation and utilization of Comprehensive Unit Plans 
(CUPS) as pait o*^ assessments during the 1989-1990 pilot year. 
STAR Data Matrices and Standnrds-Se ttinc Judgements 

The STAR is a "criterion-referenced" assessment framework (Ellett. 1990a) and each 
Teaching and Learning Component is considered an "essential" element of effective teaching a..d 
learning and each has been professionally verified by teachers and other Louisiana educators as 
an "essential" element for professional, renewable certification. Thus, a performance standard 
needs to be recommended for each component. 

The number of assessment indicators comprising each STAR component varies, though the 
number of assessments (6)(three assessors X 2 occasions; fall and spring) does not. As a result 
of a complete assessment for a year, a matrix of "I's" and "O's" will be generated for each 
assessment component. An "Acceptable" decision is recorded as a "1" and an "Unacceptable" 
decision is recor-led as a "0." Each of the assessment indicator decisions is considered a sample 
of the effectiveness of leaching and leiu-ning relative to the larger, "key ideas" reflected in a 
particuhir component. Of course, the distribution of "I's" and "O's" for a particular STAR 
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Teaching and Learning Component will predictably be somewliat different from one observaUon 
to the next and from fall to spring assessments, for an individual teacher and across teachers. 

By way of example, the 1989-1990 STAR Teaching and Learning Component of "Time" was 
defined by 8 assessment indicators. With this structure, a matrix of 48 assessment decisions 
would be generated for this Teaching and Learning Component (3 assessors X 8 assessment 
indicators X 2 occasions; fdl and spring assessments) durinjg an assessment year. Other STAR . 
components have different numbers of assessment indicators and thus, the size of the assessment 
matrix for a particular component varies from one component to the next. Sample data matrices 
for a STAR Teaching and Learning Component with 8 assessment indicators are included in 
Appendix C to illustrate hypothetical variation in "Acceptable" and "Unacceptable" perfomianccs 
relative to assessment indicators from one STAR observation and assessment occasion to the next 
for a single teacher. It should be noted that while the patterns of assessment decisions arc 
somewhat different in the two matrices, tlie percentages of "Acceptable" decisions ("I's" in the 
matrices) are the same. In viewing this matrix, it is obvious that a wide variety of patterns of 
"I's" and "O's" are possible across various STAR Teaching and Learning Components from fall 
to spring and across various teachers, observation occasions and assessment contexts. This 
assessment and decision-making structure creates flexibility in the STAR that is needed to 
accommodate a wide variety of assessment contexts. 

Given this structure, and the need to have a common "metric" for standards setting, panelists 
were asked to make a recommendation for each STAR Teaching and Leimiing Component of the 
"percentage of Acceptab le decisions that a teacher should be credited with in the complete 
assessment process in order to meet the minimum standard for professional, renewable 
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certification in Louisiana." Tliese recommended percentages for each of the STAR Teaching and 
Learning Components then, represent an initial set of "minimum performance expectations" for 
Louisiana teachers for obtaining tiie professional, renewable teaching certificate. 

Results and Discussion 

A variety of data were available as panelist's made their recommendations from one iteration 
of judgements to the next. Three recommendations for a perfomiance standard for professional, 
renewable certification were made for each STAR Teaching and Learning Component: 1) an 
initial recommendation after studying pertinent research findings and assessment indicators 
comprising a particular component; 2) a second recommendation after considerable discussion 
of the first recommendation witli other panelists in small groups; and 3) a final recommendation 
after sharing the results of the second recommendation with the enlin; group of panelists. 

Considering the number of assessment indicators comprising a particular STAR component 
in question, the research information provided, otiier panelists' recommendations and pertinent 
discussion, each panelist was asked to recommend a finiil performance standard for each 
component. The recommendation for each component translated into a numerical percentage of 
the total matrix possibilities that should be "i's" if a teacher is to meet tiie minimum standard 
(performance expectation) for professional, renewable certification in Louisiana. After each 
round of recommendations, panelists' percentages were rank ordered "^nd arithmetical means and 
standard deviations were computed. 

Table 2 presents a summary of means and standard deviations of percentages of "Acceptable" 
decisions recommended as a performance standcU-d for professional, renewable certification in 
Louisiana for each STAR Teaching and Learning Component for first, second, and final rounds 
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of recommendations. The mean scores represent ai-ithmetical averages computed across each 
round of recommendudons of all panelists. The standard deviations (S.D.'s) can be considered 
an "index of cohcsiveness" among panelists' judgements, with greater cohesiveness evident for 
smaller numbers, and more "spread" in the data and generally less cohesiveness for larger 
numbers. Standard deviation results can only be directly compared between components for 
components having the same number of assessment indicators. The numbers in Table 2 have . 
been rounded up or down accordingly to arrive at whole numbers for percentages and single 
decimal place numbers for standard deviations. 

An analysis of the means and the standard deviations of the recommended percentages 
for components in each round of iteraUons revealed that, in almost all instances, standard 
deviauons decreased from one round to tiie next round. In 18 of the 22 Teaching and Learning 
Components, the standard deviation decreased from the first round to the third and final round; 
in one component, tlie standard deviation stayed the same; in three components the standard 
deviadon increased but only negligibly (.1). Further, in 14 of the 22 components, the standard 
deviation decreased progressively in each round of iterations. 

This is consistent with the findings of Cross. Impara. Frary. and Jaeger (1984) thai 
distributions become more homogeneous when panelists are given opportunities to reconsider 
their recommendations. Increased homogeneity of distributions increases the reliability of the 

recommended standards. 

The greatest variance from the mean was seen for the six components making up 
Performance Dimension I. Preparation. Planning and Evaluation which is used in assessing the 
Compreiisnsivc Unit Plan. Greater vjmability might possibly be explained by the fact that 
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panelists had had less training and experience in assessing this Dimension and therefore were less 

certain of perfonnance expectations. 

Considering the various recommendations collectively, mean percentages for the first round 
of recommendations typically approximated 75%. This finding shows that tliesc panelists 
typically recommended that the percentage of "Vs" in most STAR Teaching and Learning 
Component matrices should be approximately 75%. Notable exceptions to this typical standard 
were for component numbers: 10 (Managing Task-Related Beha' .or. 70%); 11 (Monitoring and 
Maintaining Student Behavior, 70%); 13 (Physical Learning Environment, 83%); 14 (Lesson and 
Acdvities Initiation, 71%); 18 (Thinking Skills, 67%); and 23 (Oral and Written Communication. 
87%). The lowest recommen rlpH pe.rfnnnance standard was for STAR component 18 (Thinking 
Skills, 67%) and the hit>liest recommended standard was for STAR component 23 (Oral and 

Written Communication, 87%). 

Table 3 presents the means and standard deviations of percentages of acceptable decisions 
recommended as performance standards by teachers in comparison to other position types on tiie 
standards-setting panel. The "oilier" category includes principals, parish supervisors, college 
faculty and state department of education personnel. 

Given the "high stakes" teachers have as a group in the standards to be set, there was 
interest in whether teachers' recommendations would vary significantly from those of odier 
professionals. As indicated by the results, recommendations were very consistent. On 13 of the 
22 recommended performance standards, percentages were the same or within one point for the 
various groups; 20 of the 21 were within two poin.s of each other. Only one standard differed 
by 3 percentage points, and tiiat was for the Component of Thinking Skills, the component on 
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which pilot data showed the lowest level of STAR assessment scores (Claiidet. Hill. Ellett. & 
Naik, 1990). 

Standard deviations about the means suggest somewhat less cohesiveness in judgements 
among teaciicrs tiian among otlier groups. Variability was greater for teachers on 17 of the 22 
components. Differences were fairly small for most components, however, with the greatest 
spread shown for Component 13. Physical Learning Environment (S.D. = 6.08 for teachers; S.D. 
= 3.73 for others) and Component 4. Aids and Materials (S.D. =6.62 and 4.10 for teachers and 
others respectively). 

As the results of this standard-setting tasic were being compiled and interpreted, the 
1989-1990 version of the STAR was undergoing revision based on information from the two pilot 
years (1988-1990) of reseiu-ch and development in Louisiana. Therefore, the recommended 
standards shown in Table 2 had to be translated into the 1990-1991 revision and structure of the 
STAR that has currently been approved by the Louisiana Board of Elementary and Secondary 
Education (BESE). This revision in the STAR resulted in merging some assessment indicators 
with other components (a minor ciiangc) and deleting some assessment indicators that were 
considered somewhat "redundant." The total reducUon in the number of assessment indicators 
was from 140 (1989-1990 version) to 117 (1990-1991 version). These revisions were made in 
an attempt to make the STAR assessment process more efficient. However, consideration was 
also given to the developing research base from studies of the STAR in Louisiana and to 
maintaining the professional integrity and quality of the STAR assessment process. 

Table 4 shows Uie relationship between the structure of the 1989-1990 STAR and Uic 1990- 
1991 STAR revision and tlic application of the performance standards recommended by these 
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panelists. Included in this table lire: 1) the number of assessment indicators comprising the 
various STAR teaching and Learning Components (ttl's); 2) the size of assessment matrices for 
each component (number of assessment indicators X 3 assessors X 2 occasions; fall and spring); 
and 3) the percentage standard recommended along with the number of indicator decisions at this 
standard that should be assessed as "Acceptable." These data are shown for both the 1989-1990 
and 1990-1991 versions of the STAR. 

There are two issues reflected in the results in Table 4 when viewing recommended standards 
relative to llie number of assessment indicators comprising components in the 1989-1990 and 
1990-1991 versions of the STAR. First, with the reduction in the number of assessment 
indicators for some STAR Teaching and Learning Components, the number of allowable "O's" 
in a matrix has also been somewhat reduced. For example, Teaching and Learning Component 
#11 (Monitoring and Maintaining Student Behavior) has been reduced from 9 to 6 assessment 
indicators. This revision reduces the size of the assessment matrix (3 assessors X 9 assessment 
indicators X 2 occasions; fall and spring) for this component from 54 to 36 decisionmaking 
possibilities ("Acceptables" or "Unacceptables"). Applyin^^ the recommended standard of 70% 
to the matrices for component #11 shows tliat 16 assessment decisions could be "O's" in the 
1989-1990 version of the STAR (i.e., 54 minus 38), while only 11 assessment decisions can be 
"O's" in the 1990-1991 version of the STAR (36 minus 25) if the current, recommended standard 
is to be met. Thus, the number of "I's" required to meet the recommended standard in the newer 
(1990-1991) version of llie STAR (n=25) is less tlian what might have been required in llie 1989- 
1990 version of the STAR (n=38). However, the newer (1990-1991) version of the STAR might 
be considered somewhat "tougher" because there is less margin for "Unacceptable" decisions in 
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the total component matrix than in the 1989-1990 version of tiie STAR. 

Considering all of the Teaching and Learning Components for tiie two versions of the STAR, 
nine retained the same number of indicators and assessment matrix sizes. One component 
(Student Engagement) is not used to make certification decisions. Component #20 (Pace) was 
merged with other components in the 1990-1991 STAR revision. Twelve components were 
/ reduced in size in terms of tlie number of assessment indicators and the issues discussed above 
for component ttll (Monitoring and Maintaining Student Behavior) apply to each of lliesc 
components. Thus, a reduction in the number of assessment indicators for the STAR Teaching 
and Learning Components from 1989-1990 to 1990-1991, when viewed relative to the 
performance standards recommended by this group of panelists, makes the STAR somewhat less 
"flexible" in terms of the "requirements" that must be met for certification. Further reductions 
in the number of assessment indicato'-s in future versions of the STAR, given tiie recommei:ded 
performance standards shown in Table 3 will create even more inflexibility in the STAR as a 
data collection and decisionmaking framework. If there are further reductions, "flexibility" in 
the STAR in making assessments for certification can only be accommodated by lowering 
perfoniiancc standards for the Teaching and Learning Components. These issues are also 
important to consider in applying the STAR to the wide variety of classroom contexts (e.g, 
subject matter, class size, nature/characteristics of students, grade level, etc.) in which teaching 
and learning occur. 

Secondly, willi differences in the number of assessment indicators comprising the various 
components, and a common standard (e.g., 75%), there are fewer possibilities for allowable "O's" 
for components with a small number of assessment indicators than for those components with 
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a larger number of assessment indicators. For example, component ^^5 (Homeworlc/Home 
Learning) was reduced in terms of tiic number of assessment indicators from 4 to 3. Tiie number 
of possibilities for allowable "O's" (with a standard of 75%) dropped from 6 (24 minus 18) to 
4 (18 minus 14). Whereas, component 16 (Aids and Materials) was reduced in terms of the 
number of assessment indicators from 8 to 6. The number of possibilities for allowable "O's" 
(with a standard of 75%) dropped from 12 (48 minus 36) to 9 (36 minus 27). 

These observations about the current structure of iJie STAR as a decisionmaking framework 
might suggest a need to establish uniformity in terms of the number of "data points" in the 
decisionmaking matrices for the various Teaching and Learning Components. However, this 
requirement is not advisable since the assessment indicators have been professionally verified by 
Louisiana educators as "essential" elements of the components under which they are classified. 
The assessment indicators for the various components also have support in the existing research 
literature on effective teaching and learning. In addition, arbitrarily moving some assessment 
indicators from their curretit clr.ssification to other Teaching and Learning Components simply 
to achieve symmeUy in instrur/ient sUucture will create considerable difficulties in the conceptual 
basis of the STAR and in the professional development program to certify assessors. These 
changes would also affect the "face validity" of the STAR, the logical classification of the 
assessment indicators, and most importantly perhaps, the reliability of the various components 
for making certification decisions. 

Given these concerns, it should be noted here, that regardless of the number of assessment 
indicators comprising a particular STAR Teaching and Learning Component, the recommended 
performance standards refiect the views of a group of Louisimia educators having full knowledge 
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of the effects of selling slundards for coniponenis with differing numbers of indicators. 

Insiglit into factors influencing recommendations of panelists was gained from a 
Participant "Importance to Recommendadons' Scale completed by each panel member. Using 
a five point Liekert scale ranging from a 5 for "highly important" to a 1 for "little or no 
importance." panelists were asked to assess the degree to which eight items influenced Uieir final 
recommendations about the appropriate perfomiancc standards for each STAR Teaching and 
Learning Component. Table 5 presents tiie frequency with which participants responded. 

The importance of the STAR component for enhancing student's learning was identified 
as having had the greatest influence on Uie group as a whole with 85% ranking it highly 
important and an additional 13.3% ranking it above average in importance. Other items to which 
panelists attributed a high degree of importance were the consequences and impact on teachers 
of setting standards at various levels (66.7%. higiily important; 26.7% above average in 
importance) and the training received as part of Uie STAR professional development program to 
certify assessors (57.8%. highly important; 28.9% above average in importance). 

Feedback about the standards set by other participants and the discussion and rationale 
for standards provided by other participants appeared to have the least influence, but these factors 
were still viewed as above average or high in importance by 55.5% and 70.5% of respondents 
respectively. 

Overall the impact on students and then teaciiers appeared to have the greatest influence 
on panelists' recommendation of performance standiuds followed by knowledge of results from 
STAR reseiuch data, and last by the views of other panelist members. Panelists apparently 
viewed some teaching and learning components as "more cridcal and important for certification," 
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or as more "difficult" or "easy" given actual performance data from the 1989-1990 STAR pilot, 
than others...resulting in a reduction in tiie allowable "O's" for some Teaching and Learning 
Components relative to others. Therefore, recommended standards for some components were 
higher or lower than for other components. 

As new findings from the literature on effective teaching and learning emerge, die STAR 
may need to be lengthened to accommodate these findings and to add additional ncxibility to the 
system for decision making purposes and to assure applicability of the STAR to the wide variety 
of contexts in whicii teaching and learning occur. 

One final activity for panelists was to consider final performance standards recommended for 
experienced teachers for professional, renewable certificauon in view of a "performance 
expectation" for new teachers participating in the Louisiana Teaching Internship Program. After 
reviewing final standai-ds and entertaining considerable discussion, panelists recommended that 
a "voice vote" be taken to endorse the idea that the performance expectations for intern teachers, 
as a requirement for completing the internship program, should be die same as the recommended 
perfomiance standards for professional, renewable certification for -.xperienced teachers. The 
rationale for this recommendation was that a key purpose of the Intcrr.sMp Program should be 
to help prepare the new beginning teacher for new certification requirements, and th t the intern 
teacher should not be considered as having met the requu-ements of the Internship Program 
without some assurances that certification standards for the renewable certificate would be met. 
Panelists were asked to write an attestation affirming their agreement with this voice vote, lliere 
was little argument with this recommetidation. and all panelists agreed that the performance 
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expectation for inter,, teachers sl.oukl be ti.e same as the pertomia.ice standatxi for experienced 

teachers for renewable ceilificalion. 

Implicaiions and Recommendations 
This document has brieny described the results of an initial performance standards-setting 
task(s) with a select group of Louisiana educators for the Louisiana S>'Stem for leaching and 
learning Assessment and Review (STAR). Tiie purpose of this task was to arrive at a sot of 
recommendations for initial performance standards for experienced teachers seeking professional, 
renewable ceitification as required in the Children First Act (1988) and for beginning teachers 
to meet performance expectations as required in tiie Louisiana Teaching Inlernsiiip Program Law 
(1984). 

These recommendations were submitted to the Louisiana Department of Education (LDE) 
for review and for consideration by the Louisiana Board of Elementary and Secondary Education 
(BESE) along with a variety of issues and concerns that needed to be reviewed before a final set 
of perfonnance standards were considered and adopted by the BESE. First, and foremost, it was 
suggested that the standards recommended by this panel of Louisiana educators should not be 
considered final by any means but be viewed as a set of "benchmarks" for teachers as they 
prepared for tiieir fall. 1990 STAR assessments. A variety of factors supporting this approach 
were provided. Most of these factors had to do with the "assessment demand characteristics" 
under real conditions and possible ciianges in program implementation policies. As tiiese factors 
change, predictable changes in STAR performance levels will be evident. 

For example, it is known tiiat tiie performance distributions of assessment systems like the 
STAR change under "high stakes" conditions like tiio-^e reflected in assessments for professional. 
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renewable cenif.cauon. These Canges in perfor„,ance can effcc. no. only consideration of future 
standards levels, but tl.e tellability of assessments and the "dependabilUy" of the STAR for 
.naking certification decisions as well. In view of these predicted changes in petto.ance, it was 
pointed out that the STAR perfonnance data collected under pilot conditions did not involve 
p^paration of U,e STAR Comprehensive Unit Plan (CUP). Research studies contple.cd in the 
spring of 1990 suggest that teachers atten.pting and completing the CUP typically perfonned at 
higher levels in the actual classroom setting than teachers who did not prepare U.e CUP. This 
finding suggests that further improvetnen.s in performance levels in the fal, of 1990 relative to 
perfomtance levels evident during the 1989-1990 pilot year „tight be expected for those who 
pn^pare a CUP. Conversely, these findings may suggest that teachers who do not prepare the 
CUP as par, of the STAR assessment may obtain assessment scores that are significantly lower 
than those who do. 

The decision by U,e Board to require the CUP for all new beginning teachers in the 
Teaching Internship Program, and to not require the CUP for e.pefienced teachers for renewable 
certiflation in light of the predicted effects on performance levels tnay raise ■■assessment equity 
concerns and impact future standards-setting considerations for these two groups of teachers as 
well. Eventually the CUP. originally desigt.ed as an important, ■'refiective practice^' part of the 
assessment process for certification, may .to. be scored at all because of lack of standardization 

across teachers assessed. 

An additional, important policy-related concern that could dr»tically effect STAR 
perfonnance dismbu.ions pertains to the '■anonymity of assessors. Rese>.ch on teacher 
evaluation has clearly shown that assesstnent data are negatively skewed (inflated scores) if the 
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identity of assessors ami their individual assessment decisions are Icnown to teachers, or if the 
assessment team members are all "in-building" personnel. The social context of evaluation in 
this regard can put tremendous pressures on assessors to resist making "Unacceptable" decisions. 
The pilot process with the STAR was based on the policy of maintaining assessor anonymity and 
it was recommended tiiat this policy be followed during actual program implementation in 1990- 
1991. However, tremendous pressure has been brought to change this policy. 

This "assessor identity and scorc inflation" problem is also exacerbated if principals and 
other assessors must provide copies of raw observation notes to teachers. Pressure has been 
brought in this area as well. In the STAR pilot process, such notes were not part of the 
••evaluation,"... which is a summative decision relative to a standard...but, instead, they were 
simply used to guide the assessor in using the STAR Annotated Guide to make assessm-^.nt 
decisions. The final "evaluation" is only made when all assessment data from all three assessors 
over two assessment occasions (fall and spring) are aggregated to make the final "evaluation" 
decision relative to an endorsed perfomiance standiu-d. 

The mass of STAR data collected during the 1989-1990 statewide pilot (approximately 6.000 
assessments) was collected under conditions of "everyday practice" and with the knowledge that 
results (assessment decisions and observation notes) did not have to be shared with teachers. 
These rules were followed because of the tentative nature of the pilot and because these field 
assessments were considered "practice" assessments for assessors. However, summaries of these 
data clearly show, that under the conditions of anonymity and everyday practice, the "TAR, for 
the most part, clearly differentiates the quality of leaching and learning in Louisiana's classrooms 
(see 1989-1990 pilot results in Appendix B). Such differentiation among classrooms, and 
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"leaving room at the top," is also an important concern lor identifying teachers as "superior" for 
the Model Career Options Program as required in the Children First Act. Protecting the 
anonymity of assessors seems an important issue from these perspectives. 

The recommendations made by this group of panelists for teacher interns were made in view 
of the provision of adequate professional support from principals, master teachers and others. 
If these panelists had prior knowledge that tliere may be no support from master teachers within 
buildings (because current program logistics mitigate against this possibility), perhaps these 
panelists would have recommended a different set of standards for beginning teachers than for 
experienced teachers. With an "outside" master teacher instead of an in-building master teacher 
proposed as part of the assessment model for 1990-1991, recommendations may have been 
altered for experienced teachers as well. Belief by panelists in the kind of support that will be 
available for aii teachers is an important factor in setting perfomiance standards. 

Thus, there are essentially two sets of important issues raised with these and other possible 
changes in program implementation policies as part of the STAR assessment: 1) changes 
expected in STAR performance distributions in the fall and spring of 1990-91 and their 
implications for future standiu-ds-setting considerations; and 2) the more global concern of policy 
changes such as whether ciu-eful, reflective practice should be part of a system in Louisiana to 
renew the professional teaching credential. The first concern, perhaps, awaits additional empirical 
infonnation from assessments completed in 1990-91. The second concern is whether Louisiana's 
system will stay "in step" with future generations of assessment practices and methodologie;; 
being proposed and/or piloted by those wo.xing in the teacher certification field (e.g., the 
National Board of Professional Teaching Standards).. .or whetlier policy clianges will affect not 
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only standards but result in the replication of systems that have been developed and received 
"mixed blessings" in other states. The assessment of careful planning and reflective practice as 
an indication of content and pedagogical knowledge, for example, is currently viewed by the 
NBPTS and experts in the assessment field as an important assessment concern (NBPTS, 1990). 

The standards recomi.iended in this study were the "best" available based on the views 
of a panel of knowledgeable Louisiana educators consisting of a majority of classroom teachers. 
It was recommended that these standards be temporarily endorsed by BESE as appropriate 
"benchmarks" to guide teachers as tliey prepare for STAR assessments during the fall of 1990. 
It was also strongly recommended that standards be carefully reviewed and additional 
recommendations be made in the fall of 1990 and the spring of 1991 after data were analyzed 
that were collected under "real" conditions with final program implementation policies put into 
effect. Because of tlie tentative nature of the standards, it was further recommended that final 
decisions about professic.al, renewable certification not be made until the late spring of 1991, 
and that no teacher be considered "in remediation" until the fall of 1991. This procedure would 
allow those teachers who were initially randomly selected by the LDE for assessment to have a 
maximum of three years before any consideration co"ld be given to invalidating the professional 
teaching credential. This seemed reasonable given the temporary nature of currently 
recommended standards reflected in this report and given the fact that some teachers would have 
much more time to prepare for tlie assessment process than other teachers. 

Unfortunately, because teacher evaluation and standards-setting, must operate witliin a 
political and highly emotional arena, tiic most sound approach does not always take precedence. 
Responding to a great deal of pressure, the Louisiana Department of Education and the Board 



erIc 



28 



27 

of Elementary aiid Secondary Educadon made the decision to set final standards in December 
of 1990. 

An external advisory committee of six consultants from within and outside tiie state met 
to review standards-setting issues, STAR research data and assist in establishing the final 
standards. After four days of meeting together the committee made a variety of recommendations 
relative to standards-setting concerns, decision-making models and elements of program 
implementation. A second set of three "external" consultants witii extensive experience in the 
field of assessment reviewed these recommendations and suggested additional factors to be 
considered. Tlie collective recommendations of both consultant groups were used in designing 
and implementing the Fall 1990 STAR standards-setting task. 

Upon the recommendation of the external advisory groups, the standards-setting panel 
consisted entirely of classroom teachers, ten of whom iiad served on the original "expert panel" 
that set initial STAR benchmarks and the remainnig group of 15 who had been assessed in the 
Fall 1990 STAR assessment. As part of a two-day workshop teachers considered various models 
and procedures for making assessment decisions, and teachers were able to review some of the 
fall data collected under high stakes conditions. These results showed "higher" scores under real 
versus pilot conditions. Higlier scores have been interpreted to reflect actual improvement as 
well as artificial score inflation. (See Appendix D for a comparsion of pilot and fall data.) 

Benchmarks established for each Teaching and Learning Component during the Spring 

of 1990 remained unchanged. However, teacher panelists recommended two substantive changes 

in the STAR decision-making model: 

1. requiring that teachers achieve tlie benchmark on any 13 of the teaching 
components rather than all 15 as proposed in the original model, and 
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2. using a combined "conjunctive" and "compensatory" model sucli tiiat the required 
numbers of Teaching and Learning Components must be at or exceed the 
"bencinnark" standards ajid an overall assessment score of 75% siiould also be 
met. 

The BESE accepted these recommendations, and standards were set accordingly. Since 
that time a number of ramifications have brought into question the decision to depart from the 
original criterion-referenced decision-making model. Not only do tiie changes diminish tiie 
constiiict validity of the STAR and llie importance of each component iji leaching and learning 
but it has also resulted in a number of "scoring anomalies," liius raising "equity" concerns, lliese 
concerns just reinforce the view that standards- setting is an ongoing process. 

In future years, as LTIP and LTEP are implemented with all teachers, the LDE and BESE 
will need to revisit established performance standards for the STAR in view of: 1) performance 
data collected in Louisiana's classrooms under "real" conditions (a summary of the results of the 
completed evaluation of 1,701 teachers during the 1991-92 school year is included in Appendix 
D); 2) ramifications of policy decisions, ; 3) research results emerging from studies of the 
STAR and effective teaching and learning; and 4) tlie value that Louisiana educators and citizens 
place on particular elements of effective teaching, professional practice, and children's learning. 

The following are just a few of the issues to be considered as standards are reviewed and 
revised: 

Ilow can wc arrive at a set of slandiuds for STAR that 

(1) assure a "reasonable" level of competence at certification levels, 

(2) will protect the "integrity" of new important skills such as content 



structure/emphasis and thinking skills. 



(3) 



will drive staff development and professional improvement, 
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(5) will send and protect the "accountability" message desired by tlic 
Governor's Office, BESE, tiie legislature and the general public, 

(6) will not create "supply and demand" problems in the teacher work force, 

(7) will not "overload" principals with remediation work, 

(8) will be clear and understandable to teachers, principals, policymakers and 
the general public, 

(9) allow for "revisiting" and "adjusting" after a sufficient number of teachers 
have been assessed both Fall and Spring, 

(10) will reflect what is emph-ically known about STAR validity and reliability 
characteristics, 

(11) discourage score "solution," "corruption," and "inflation," realizing that no 
single model is likely to be able to completely control this "noise" in the 
System, 

(12) will be useful in detenniniiig/examining "significant progress," 

(13) will be "non-discriminatory" against particular classes of teacher (e.g., 
race, gender, grade level, etc.), 

(14) will reinforce the concepts of "criterion-referenced" assessment and 
"banking" for certification; and criterion-referenced assessment for MCOP 
(no banking), 

(15) will allow the SDE with available monies to successfully manage the 
program (e.g., number of teachers in remediation ("carryovers"), number 
of available assessors, etc.) 

(16) will be sensitive to the statistical "dependability" of decisions and will 
minimize "false negative" and "false positive" decisions (this relates to 
reliability concerns in fflO above), and 

(17) will be conceptually consistent with the "holistic" construct validity of tlie 
STAR? 

As stated in the introduction to this paper, setting performance standards for systems like 
the STAR is not an easy task. When considered in light of a volatile socio-political context, the 
task becomes even more complex. 
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TABLE 1 34 



LTIP AND LTEP INITIAL STANDARDS-SEITINGS 

COMMITTEE 

PARTICIPANT DEMOGRAPHIC INFORMATION 



Sex: 



Ellinicitv: 

Black 



Male 32.6% 
Female ^7.4% 



39.1% 



Wliiie 60.9% 

Average Age; ^-^••^ "^^^^^ 
Current Posilion: 

Teachers 56.5% 

Principals ^-^^ 

Assislanl Principals 17.4% 
College Facully 
OUicr 

Average Number of Years Employed in Current 11.89 Years 
Position 

School Level in Which Currently WorkinK: 

Eariy Childhood 2.3% 

Elementary 41.9% 

Secondary/High School 18.6% 

CoUege 7.0% 

Multiple School Levels ^-T^" 



Average Years Experience As An Educator: 

Public Schools 18.4 Years 

Private Schools 0-6 Years 

Teachers' Average Years Teaching Experienc e: 

Public Schools 17.28 Years 
Private Schools 1.00 Years 

Iliglicst Degree Earned: 

Bachelor 0.0% 

Master 13.0% 

Master Plus Graduate Hours 30.4% 

Plus 30 or Specialist 47.8% 

Doctorate 8.7% 
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TABLE 2 

Summary of Means and Standard Deviations of Percentages 
of "Acceptable" Decisions Recommended as a Performance 
Standard for Each STAR Teaching and Learning Component for 
Tliree Rounds of Panelists' Recommendation 



STAR Teaching and Lcaiiiing CotnDoncnls Mean' and S.D.''bv Rounds 







i 


i 


2 


2 


3 


3 


1. 


Goals and Objectives 


68 


12.5 


73 


8.6 


75 


6.3 


2. 


Tcaciiing Mediods and Learning Tasks 


73 


6.3 


74 


6.3 


75 


5.2 


3. 


Allocated lime and Content Coverage 


74 


8.3 


75 


5.7 


75 


4.9 


4. 


Aids and Materials 


75 


8.4 


77 


7.2 


76 


5.5 


5. 


Homework 


74 


9.3 


75 


8.3 


75 


5.5 


6, 


Formal Assessment and Evaluation 


74 


8.5 


72 


8.6 


74 


5.6 


7. 


Time 


77 


5.9 


74 


6.9 


75 


4.1 


8. 


Classroom Routine 


77 


6.7 


76 


7.3 


76 


4.3 


9. 


Student Engagement' 














10. 


Managing Task-Related Behavior 


67 


7.2 


68 


7.1 


70 


6.3 


11. 


Monitoring and Maintaining 


70 


7.4 


69 


6.4 


70 


5.9 




Student Behavior 














12. 


Psychosocial Learning Environment 


76 


5.9 


77 


6.0 


77 


5.5 


13. 


Physical Lcamuig Environment 


86 


5.1 


85 


5.2 


83 


5.2 


14. 


Lesson and Activities Initiation 


67 


7.9 


69 


6.8 


71 


4.8 


15. 


Teaching Melliods and Learning 'I^sk 


75 


5.6 


75 


5.7 


74 


4.5 


16. 


Aids suid Materials 


76 


5.0 


76 


5.4 


75 


5.1 


17. 


Content Accuracy and Emphasis 


74 


5.9 


75 


5.0 


75 


3.9 


18. 


Thinking Skills 


65 


10.5 


65 


9.7 


67 


8.2 


19. 


Clariricalion 


75 


4.2 


76 


4.0 


75 


4.3 


20. 


Pace 


75 


5.4 


75 


6.0 


74 


5.4 


21. 


Monitoring Learning Tasks 


74 


4.8 


74 


4.8 


75 


3.9 




and Informal Assessment 














22. 


Feedback 


73 


5.2 


73 


5.5 


74 


4.9 


23. 


Oral and Written Communications 


87 


6.3 


88 


6.0 


87 


4.9 



'Mean = Arilhmclical average coinpuled over all panelists* i)crforniancc suiiuhircis recoinmcndalions 
^S.D. = Slandard Deviation of scores rccoi^imendcd by panelists 

'Student Engagement standard was not recommended because lliis component is not used to make cerlificalicn 
decisions 
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TABLE 3 

Comparison of Means and Standard Deviations of Percentages 

of "Acceptable" Decisions Recommended as a Performance 
Standard for Each STAR Teaching and Learning Componet by 
Teacher Panelists and other Position Types 



STAR TcacliiiiR aiid Learning CoinponciU 


Teachers (n=26) 


Others (n=18) 






Mean* 


S.D.'' 


Mean 


S.D. 


1. 


Goals and Objectives 


75 


6.65 


75 


5.91 


2. 


Teaching Mclliods and Learning Tasks 


75 


5.49 


75 


5.10 


3. 


Allocated Time and Content Coverage 


74 


5.66 


76 


4.08 


4. 


Aids and Materials 


75 


6.62 


76 


4.10 


5. 


Homework 


75 


6.22 


75 


4.99 


6. 


Formal Assessment and Evaluation 


74 


6.41 


74 


4.05 


7. 


Tunc 


75 


3.29 


76 


5.20 


8. 


Classroom Routine 


75 


4.51 


77 


4.35 


9. 


Student Engagement' 










10. 


Managing Task-Related Dcimvior 


69 


5.78 


71 


6.95 


11. 


Monitoring and Maii.tainlng Student 


69 


6.20 


71 


5.39 




Deliavior 










12. 


Psycliosocial Learning Environmctu 


76 


6.43 


78 


4.10 


13. 


Pliysical Learning Environment 


82 


6.60 


83 


3.58 


14. 


Lesson and Activities 


71 


4.48 


72 


5.16 


15. 


Teaching Mclliods and Learning Task 


74 


3.93 


74 


5.35 


16. 


Aids and Materials 


75 


5.59 


76 


5.02 


17. 


Content Accuracy and Emphasis 


74 


3.56 


76 


4.26 


18. 


Tiiinking Skills 


65 


8.26 


68 


8.15 


19. 


Clarification 


74 


4.36 


76 


4.04 


20. 


Pace 


73 


6.14 


75 


4.17 


21. 


Monintoring Learning Tasks and Informal 


74 


4.22 


75 


4.07 




Assessment 










22. 


Feedback 


73 


5.60 


74 


3.98 


23. 


Oral and Written ConiniunicaLions 


86 


6.08 


87 


3.73 



'Mean = Arithmetical avcinjje coiuimlal over all pajiclLsls' iwrronnancc .slimdurds rccoinnicndalions 
•"S.D. ■■= Slaiulard Deviation of scores recommended by panelists 

*Sl'jdenl Engagement siiindurd was not raonunendcd bwanse this component is not used to make 
cerlificalion decisions 
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TABLli 4 

Compjirisoa of llic Nunibcr of Assessment Indicators for 1989-1990 
STAR and 1990-1991 STAR Teaching and Learning Components, 
Fall and Spring Assessment Matrix Sizes, and Recommended 
Perfonnance Standard Percentages and Numbers 



STAR Teaching and Learning Components 



1. Goals and Objectives 

2. Tcacliing MctJiods and Learning Tiisks 

3. Allocated lime and Content Coverage 

4. Aids and Materials 

5. ^ Homework ("Home Learning)"^ 

6. ' J;ormal Assessment and Evaluation 

7. Time 

8. Classroom Routine 

9. Student Engagement^ 

10. Managing Task-Related Bcliavior 

1 1 . Monitoring and Maintaining 
Student Dcliavior 

12. Psycliosocial Learning Environment 

13. Pliysical Learning Environment 

14. Lesson and Activities Initiation 

15. Twicliing Metljods and Learning Task 

16. Aids and Materials 

17. Content Accuracy and Emphasis 

18. Tlmiking Skills 

19. Clarification 

20. Pace^ 

21. Monitoring Learning Tasks 
and Informal Assessment 

22. Feedback 

23. Oral and Written Comnmnications 





1989-1990 


1990-1991 






in S 


MQ QLfU\ 

iVlo /o[u ) 


ili 5 


Mo 




6 


JO 










6 


JO 


n<r01\ A 


OA 






4 




IJ\iQ) *4 


OA 








< 
J 




A 


OA 


76(18) 


4 


24 


75(18) 3 


18 


75(14) 






7 


42 74(31) 


7 


42 


74(31) 




0 
0 




0 


JO 


75(27) 




4 


24 76(18) 


4 


24 


76(18) 


6 


36 


70(25) 6 


36 


70(25) 


- 


9 


54 


70(38) 6 


36 


70(25) 




12 


72 


77(55) 10 


60 


77(46) 




A 


24 


83(20) 3 


18 


83(15) 




10 


60 


71(43) 8 


48 


71(34) 




6 


36 


74(27) 6 


36 


14{21) 






8 


48 75(36) 


6 


36 


75(27) 


7 


42 


75(32) 6 


36 


75(27) 




11 


66 


67(44) 1 1 


66 


67(44) 






5 


30 75(33) 


4 


24 


75(18) 




3 


18 74(13) 










6 


36 75(27) 


6 


36 


75(27) 




4 


24 74(18) 


4 


24 


74(18) 




4 


24 87(21) 


4 


24 


87(21) 



'ill's = Number of asscssinciil indicalors comprising a STAR Tcaciiing arid LcariiinB Component; 
MS = Matrix sizo; 

%(//) = Pcrcciiliific sUindard rccomnicnJcd and number of assessment iiulicalor decisions iJiat must bo "I's" 
(Acceptable) 
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TABLE 5 

Frequency of tlie Degree to Wliich Items 
Influenced Final Recommendations of 
Perfonnaiice Standards for the STAR Teaciiing 
and Learning Components for Experienced 
Teaciiers for Renewable Certification 





Items Influencinc Recommendations 


Dceree of Importance' 
by Percentage of Respondents Cn=45) 






i 


2 


3 


4 


5 


1. 


llie blAR professional development 
program to certify assessors. 


0.0 


8.9 


4.4 


28.9 


578 


9 


juireci experience in tiie iieici ouserving 
teachers with the STAR. 


0.0 


2.2 


11.1 


46.7 


4.0 


J. 


Knowledge of results from STAR research 
data presented in this program. 


0.0 


2.2 


22.2 


37.8 


37.8 


4. 


Knowledge of STAR reliability and decision- 
making concerns presented/discussed in this 
program. 


0.0 


2.2 


17.8 


42.2 


37.8 


5. 


Feedback about standards set by otJicr 
participants in tills program. 


2.2 


13.3 


28.9 


22.2 


33.3 


6. 


Discussion/rationale for standards provided 
by other participants. 


0.0 


15.9 


13.6 


43.2 


27.3 


7. 


The consequences and impact on leaciiers of 
selling standards at various levels. 


0.0 


4.4 


2.2 


26.7 


66.7 


8. 


The importance of the STAR component for 
enhancing students' learning. 


0.0 


0.0 


2.2 


13.3 


84.4 



'Importance scale 



1 = Little or No Importance 

2 = Some Importance 

3 = Average Importance 

^4 ss Above Average in Importance 
ERIC= Iliglily Important 



APPENDIX B 



Sample Set of STAR Research Results Provided 
To Each Standards-Setting Panelist 
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TABLE 1 

PcrccnUigc of Maxiinmii Possible for Tcaclviiig and Lcaniing 
Coiiiponcnls for Each Dunciision of Uic STAR 
(Indicalors = 108) 
(N - 5720) 



TEACHING AND LEARNING COMPONENTS 



// of 
Indicators 



Mnxiinuin 
Possible 



% of 

Maximum 



PERFOUMANCE DIMENSION li: 
CLASSROOM AND REIIAYIOR MANAGEMENT 

A. Time 

D. Ciassrooni Routines 

C. Student EiigagciMcnt 

D. Managing Task-Rclalcd Dcliavior 

E. Moiiitoruig and Maijilainiiig Student Deiiavior 



8 
4 
1 
6 
9 



43.784 
21.892 
5,473 
32.838 
49,257 



72.39 
74.17 
36.87 
48.48 
54.21 



PERFORMANCE DIMENSION lU: 
LEARNING ENVIRONMENT 



A. 
D. 



Psychosocial 
Physical 



12 
4 



65.676 
21,892 



66.^10 
88.03 



PERFORMANCE DIMENSION IV: 
ENHANCEMENT OF LEARNING 



A. 
D. 
C. 
D. 
E. 
V. 
G. 
II. 

I. 
J. 



Lesson and Activities Ijiitiatioii 

Tcaclung MctJiods and Lcaniuig Task5 

Aids aiid Materials 

Conlcnl Accuracy ajid Einpluisis 

'IliinkiJig Skills 

Ciaiiricatioii 

Pace 

Monitoring Lcaniijig Tasks ajid 
Infonnni Assessment 

Feedback 

Oral and Written Conununicallon 



10 
6 
8 
7 
11 
5 
3 

6 
4 

4 



54;730 
32.838 
43.784 

38311 
60,203 
27,365 
16,419 

32,838 
21,892 
21,892 



34.45 
58.64 
61.78 
49.14 
21.56 
54.28 
58.02 

43.15 
33.22 
94.70 
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TEACHING AND LISAKNING COMI'ONENTS . 




Percent 
Acceptable 




Percent 
Unacceplabic 








N 


DT 


ET 


N 


DT 


ET 


1). 


CONTENT ACCURACY AND 
EMPHASIS 










1 




I. 


SludcnUs arc given oppoilimilics lo learn ai 
more lliaii one coEnilivc aiici/or 
pcrfoniiajicc level. 


32.0 


31.3 


32.3 


88.0 


68.7 


67.7 


2. Einpliasiics Uic value and imporuincc of 
topics and acliviilcs. 


19.2 


18.6 


19.0 


80.8 


81.4 


IJl.O 


3. 


Conleal knowledge is nccurale arid up-lo- 
dale. 


93..5 


90.1 


94.5 


6.5 


9.9 


5.5 


4. 


Conlenl knowledge is logical. 


87.2 


82.8 


88.1 


12.8 


17.2 


11.9 


5. 


Dircclions aJid explanations rchlcd lo 
lesson conlenl and/or learning lask5 are 
effeclivc. 


57.1 


53.1 


57.3 


42.9 


46.9 


42.7 


6. 


r.^^Miini r>inniciii<: nf conlenl knowledge 
csseniiai cicnicuii ui v-umi-m « . 

and/or performance lasks ate emphasized. 


28.7 


25.3 


28.9 


71.3 


74.7 


71.1 


7. 
15. 


. r>r uniiiic nf (lifficullv arc 
Polcnlial areas or poinrs oi uuuvuu/ 

emphasized iluoughoul llic lesson. 
TlllNiaNG SIULLS 


26.2 


22.6 


26.3 


73.8 


77.4 


73.7 


1. 

2. 


AssociaUons are lauglil and used in 
learning. 

ti.vniunc chiflrni<; in dcvelopinR concepts. 


26.3 
26.1 


26.1 
25.0 


26.6 
26.7 


73.7 
73.9 


73.9 
75.0 


73.4 
73.3 


3. 


Involves sludenls in dcveiopijig principles 
and/or rules. 


15.3 


H.J 


1 J. J 


84.7 


85.5 


84.7 


4. 

5. 
6. 


Encourages sludcnLs lo lliink of and rccaU 
cxa;nples from lliei/ own experiences. 

Encourages sluden's lo use nieiiuU imagery. 
Asks a variety of queslions. . 


22.7 
13.9 
29.3 


on A 

15.4 
31.3 


13.6 
28.8 


77.3 
88.1 
70.7 


77.6 
84.6 
68.7 


77.3 
86.4 
71.2 


7. 


Wail lijnc is used lo cnliance siudenl 
learning. 


38.2 


36.4 


37.7 


61.8 


63.6 


62.3 


8. 


Encourages critical analysis and/or problem 
solvijig. 


18.4 


16.6 


18.4 


81.6 


83.4 


81.6 


9. 
10. 


Encourages sludcnlii lo elaborate, cxleiid or 
criliquc llicir own or oUicr sludenls 
responses. 

Encourages crealive llunking. 


18.5 
14.8 


■ 16.0 
15.4. 


18.2 
14.2 


81.5 
85.2 


84.0 
84.6 


81.8 
85.8 


U. 


Provides opporlunidcs for Uic extension of 
learning lo )iew conlcxts. 


13.6 


14.3 


13.1 


86.4 


85.7 


86.9 
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TEACHING & LEARNING COMPONENT 
IV E: THINKING SKILLS 




TEACHING & LEARNING COMPONENT 
IV E: THINKING SKILLS 
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1988-1989 Research 
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Generalizability Coefficients for the STAR Teaching/Learning Components 



Teaching/ 

Learning 

Component 



G-Coefficient: 
Principal and 
External Assessor 



G-Co9fticient ' 
Principal, External Assessor 
and Master Teacher 



#7 Time .598 
# 8 Classroom Routines .525 

#10 Managing Task-Related 

Behavior .645 

#11 Monitoring/Maintaining 

Student Behavior .723 

#12 Psychosocial Learning 

Environment .726 

#13 Physical Learning 

Environment .631 

#14 Lessons/Activities 

Initiation .664 

#15 Teaching Methods .577 

#16 Sequence/Pace .521 

#17 Aids and Materials .614 

#18 Content Accuracy/ 

Emphasis .660 

#19 Thinking Skills .732 

#20 Clarification .447 

#21 Monitoring Learning 

Activities/informal .596 
Assessment 

#22 Feedback .625 

#23 Oral/Written 

Communication .130 



4() 



.643 
.577 

.700 

.775 

.789 

.695 

.722 
.630 
.576 
.682 

.728 
.807 
.497 

.651 

.691 

.147 



19Q9-1990 Research . 

Gonorallzablllty Coofflclonts for llie STAR Teaching and Learning Components 

Teaching and Loar„lng Compononts Sl^al 

Extornal Assessor Assessor and 

Master Teacher 

PERFORMANCE DIMENSION II: 
CLASSROOM AND BEHAVIOR MANAGEMENT 

A. Time 

B. Classrcwni Roulirios 

D. Managing Task-Relaled Behavior 

E. Monitoring and Maintaining Student Behavior 



0.223 
0.441 
0.595 
0.561 



0.292 
0.540 
0.683 
0.655 



PERFORMANCE DIMENSION III: 
LEARNING ENVIRONMENT 

A. Psychsoclai 0.4 61 0.5,57 

B. Physical 0.30 0.391 

PERFORMANCE DIMENSION !V: 
ENHANCEMENT OF LEARNING 

A. Lesson and Activities Initiation 0.397 0.497 

B. Teaching Motliods and Learning Tasks 0.616 0.702 

C. Aids and Materials 0.386 0.463 

D. Content Accuracy and Emphasis 0.383 0.''^3 

E. Thinking Skills 0.433 . 0.526 

F. Clarificayon 0.327 0.419 

G. Pace 0.268 0.355 

H. Monitoring Learning Tasks and Inlormal 0.560 0.647 
Assessment 

I. Feedback 0.370 0.462 
J. Oral and Written Communication 0.340 0.435 



o 
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APPENDIX C 

Hypothetical STAR Assessment Matrices 
for One Teacher for One Teaching 
and Learning Component 
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Hypothetical Data Matrices for a STAR Teaching and Learning Component 
for One Teacher for an Assessment Year for One Teaching and Learning Component 

Comprised of Eight Assessment indicators 



Fall Assessment 
Indicators Pf MT Ext 



3 
4 
5 
6 
7 
8 



0 • 0 



0 
1 
1 
1 
1 
0 



1 
1 
1 
1 
0 

1 



1 

1 

0 
0 
0 

1 



5 6 
Total "1's" = 16 (67%) 



Spring Assessment 
Indicator P MT 



1 
2 
3 
4 
6 
6 
7 
8 



0 



0 



1 
0 

1 
1 

0 

1 
1 

0 



Total "I's" = 16 (67%) 



GRAND PERCENTAGE = 67%'' 



P = Principal; MT = Master Teacher; Ext = External Assessor 
oprand Percentage = Sum of "1 's" for the two Matrices Divided by 48 Possibilities 
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APPENDIX D 



Summary of STAR Dala Aiialys 
Evaliialions ComplclccI 
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Summary of Percentages of Teachers at or Exceeding "Benchmarks" for 
STAR Teaching and Learning Components from Spring, 1990 
"Pilot" (Single Assessor) Assessments and Fail. 1990 "Real" (AU Tlu-ce Assessors) Assessments 



Tcacliing aiid 
Learning Components 


9nnncr 1000 

(n = 5.720) . 


Fall 1 Q9n 
(n = 2,587) 




65 


92.2 


2. Routines (/6) 


oy 




3. Managing Task-Rclaicd Behavior (/Uj 




M 0 


4. Moditoring and Maiiuaining 
Student Behavior (70) 




S'' 5 


5. Psychosocial Learning Environment (77) 


34 


92.2 


6. Physical Learning Environment (S3) 


68 


94.9 


7. Lesson and Aciividcs Inidadon (71) 


7 


58.5 


R Teaching Mcdiods/Leamins Tasks (74) 


34 


90.8 


y. /^das ana xviaicndib \ 


49 


90.1 


10 Content Ac curacy/Em oh as is (75) 


11 


67.7 


IL ITiinking Skills (67) 


6 


38.4 


12. Clarification (75) 


40 


91.0 


13. Monitoring Learning Tasks and 
Informal Assessment (75) 


20 


84.8 


14. Feedback (74) 


22 


73.1 


L*. Oral and Written Conununicadon (87) 


87 


97.8 



Recommended "Benchmark" 
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Summary of STMK Data Analysis Results for TEP Teachers 50 
Completed on Existing SDE Data rile as of February 22, 1991 

(0=^1,701 teachers) 



1. "Satisfactory" Decisions 

if Components Passed # Teachers 

13 151 
lA 377 
15 9_91 

TOTAL = 1519 

Percentage of teachers "passing" STAR with current requirements 
= 89.3% 

* Note: The 1519 teachers are exactly the same teachers with or 
without the "compensation score" of 395. Therefore the 
compensation score does little to contribute to the decision to 
certify . 

2. "Superior" Decisions 

a Components ."assed // Teachers 

14 82 

15 47 

TOTAL = 12 9 

Percentage of teachers qualifying for MCOP relative to total number 
of teachers assessed in file =7.6% 'lumuer 

* Note: The number of teachers with a Master degree and at least 
7 years teaching experience = 254. The 129 "qualifiers" for MCOP 
meeting standards is equal to 50.8% of the sample of 254 teachers 



3. The percentage of teachers "passing" all 15 components at the 
current benchmark standards is 58.3% (991/1701). 

4. The percentage of teachers "passing" all 15 components n^ hh« 
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5. The percentages of teachers that "pass" either 13 or 14 
components at current benchmarks and "fail" selected components are 
as follows: 

"Pass" 13 Components 

Thinking Skills (122 of 151 "fail") (00. Gi;) 

(28 of 151 "fail") (18.5%) 
(65 of 151 "fail") (43.0%) 
(45 of 151 "fail") (29.8%) 



Feedback 

Lesson Initiation 



Content Accuracy/ 
Emphasis 



" Pass" 14 Components 

Thinking Skills (267 of 377 "fail") (70.8%) 

(23 of 377 "fail") (6.1%) 
(36 of 377 "fail") (9.5%) 
(21 of 377 "fail") (5.6%) . 



Feedback 

Lesson Initiation 



Content Accuracy/ 
Emphasis 

* Note: A total of 309 teachers (73.7%) in these two groups of 
"certifiable" teachers "fail" to meet the current benchmarks 

for the STAR Thinking Skills Component. 

* Note: The percentage of teachers "failing" to meet the current 
benchmark standard for the Thinking Skills component of all three 
groups of "certifiable" teachers (13, l4 and 15 "passed") is 25.6%. 



6. The percentage of teachers that "pass" either 11 or 12 
components at current benchmarks and "fail" selected components are 
as follows; 

"Pass" 12 Components 

Thinking Skills (63 of 69 "fail") (91.3%) 

. (24 of 69 "fail") (34.0%) 

(49 of 69 "fail") (71.0%) 

(39 of 69 "fail") (56.5%) 



Feedback 

Lesson Initiation 
Content Accuracy/ 



* Note: If the "passing" model for certif icalion is lowered to 
"any 12" components/ tlie overall "pass" rate would increase from 
09.3% to 93.4%. 
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Pass" 11 ComponentsS 
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II 



Thinking Skills 



(33 of 36 "faii'M 



(91.7%) 



Feedback 



(16 of 36 "lail") 



(44 .4%) 



Lesson Initiation (29 of 36 "fail") (80.6%) 

Content Accuracy/ (24 of 36 "fail") (66.7%) 
Emphasis 

* Note: If the "passing" model is lowered to "any 11" components, 
the overall "pass" rate would increase from 93.4% (above example) 
to 95.5%. 

* Note: If the "passing" model is lowered to "any 11" components, 
the overall "failure" rate on Thinking Skills for those passing 
fewer than 15 components would be 76.6%. 

7. An analysis of a file of 7, 787 complete fall, 1990 STAR 
assessments showed that 4,727 teachers (60.7%) were at or exceeded 
current benchmarks on 13 or more components. A similar analysis 
showed that 198 teachers (2.54%) were at or exceeded current 
benchmarks for MCOP on 14 or 15 components. 



8. The following percentages are "failure" percentages for each 
STAR teaching and learning component based on analyses of the file 
of 1,701 complete assessments when compared to existing "benchmark" 
standards for each component: 

STAR Component % "Failures" 

Time 1.6% 

Routines 0.7% 

Managing Task- 3.8% 
Related Behavior 

Monitoring/Maintaining 6.4% 



Behavior 



Psychosocial Learning 
Environment 



1.2% 



Physical Learning 
Environment 



0.8% 



Lesson Initiation 



14.8% 



Methods and Tanks 



1.9% 
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Aids and Materials 2.1% 53 

Content Accuracy /Emphasis 11.3% 

Thinking Skills 32.8% 

Clarification 3.2% 

Monitoring LearningTasks/ 4.2% 
Informal Assessment 

Feedback 8.7% 

Oral and Written 0 .4% 
Communication 

9. The following percentages show "pass" rates for each STAR 

teaching and learning component for fall, 1990 and spring 1991 
assessments for the sample of 1,701 teachers for each component. 

STAR Component Fall % Sprinq% 

Time 9 6. S 9 9.6 

Routines 98.1 99.5 

Managing Task-Related 92.0 98.2 
Behavior 

Monitoring/Maintaining 90.1 96.7 
Behavior 

Psychosocial Learning 96.3 99.5 
E.nvironment 

Physical Learning 98.5 99.3 
Environment 

Lesson Initiation 70.6 92.8 

Methods and Tasks 91.9 98.5 

Aids and Materials 93.9 99.1 

t 

Content Accuracy/ 79.5 96.9 
Emphasis 

Thinking Skills 'IS. 7 88.4 

Clarification 92.1 98.2 
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Monitoring Learning Tasks/ 91.9 98.4 
Informal Assessment 

Feedback 85.0 97.9 

Oral and Written 99.4 99.6 

Communication 
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